Exploiting inherent relationships in RNN architectures

We provide the relationship between the learning rate and the slope of a nonlinear activation function of a neuron within the framework of nonlinear modular cascaded systems realised through Recurrent Neural Network (RNN) architectures. This leads to reduction in the computational complexity of learning algorithms which continuously adapt the weights of such architectures, because there is a smaller number of independent parameters to optimise. Results are provided for the Gradient Descent (GD) learning algorithm and the Extended Recursive Least Squares (ERLS) algorithm, using a general nonlinear activation function of a neuron. The results obtained degenerate into the corresponding results for single RNNs, when considering only one module in such cascaded systems.